Fabricating cooled electronic system with liquid-cooled cold plate and thermal spreader

ABSTRACT

Methods are provided for facilitating cooling of an electronic component. The methods include providing a liquid-cooled cold plate and a thermal spreader associated with the cold plate. The cold plate includes multiple coolant-carrying channel sections extending within the cold plate, and a thermal conduction surface with a larger surface area than a surface area of the component to be cooled. The thermal spreader includes one or more heat pipes including multiple heat pipe sections. One or more heat pipe sections are partially aligned to a first region of the cold plate, that is, where aligned to the surface to be cooled, and partially aligned to a second region of the cold plate, which is outside the first region. The one or more heat pipes facilitate distribution of heat from the electronic component to coolant-carrying channel sections of the cold plate located in the second region of the cold plate.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. Ser. No. 13/102,200, filedMay 6, 2011, entitled, “Cooled Electronic System with Liquid-Cooled ColdPlate and Thermal Spreader Coupled to Electronic Component”, which waspublished Nov. 8, 2012, as U.S. Patent Publication No. 2012/0279686 A1,which is hereby incorporated herein by reference in its entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with Government support under Contract No.DE-EE0002894, awarded by the Department of Energy. Accordingly, the U.S.Government has certain rights in the invention.

BACKGROUND

As is known, operating electronic devices produce heat. This heat shouldbe removed from the devices in order to maintain device junctiontemperatures within desirable limits, with failure to remove heateffectively resulting in increased device temperatures, potentiallyleading to thermal runaway conditions. Several trends in the electronicsindustry have combined to increase the importance of thermal management,including heat removal for electronic devices, including technologieswhere thermal management has traditionally been less of a concern, suchas CMOS. In particular, the need for faster and more densely packedcircuits has had a direct impact on the importance of thermalmanagement. First, power dissipation, and therefore heat production,increases as device operating frequencies increase. Second, increasedoperating frequencies may be possible at lower device junctiontemperatures. Further, as more and more devices are packed onto a singlechip, heat flux (Watts/cm²) increases, resulting in the need to removemore power from a given size chip or module. These trends have combinedto create applications where it is no longer desirable to remove heatfrom modern devices solely by traditional air cooling methods, such asby using air cooled heat sinks with heat pipes or vapor chambers. Suchair cooling techniques are inherently limited in their ability toextract heat from an electronic device with high power density.

BRIEF SUMMARY

In one aspect, the shortcomings of the prior art are overcome andadditional advantages are provided through the provision of a method offacilitating dissipation of heat from at least one electronic component.The method includes providing a cooling apparatus configured to coupleto at least one electronic component to be cooled, the cooling apparatushaving a larger footprint size than the at least one electroniccomponent to be cooled, and the cooling apparatus including: aliquid-cooled cold plate comprising a thermally conductive material witha plurality of coolant-carrying channel sections extending therein, theliquid-cooled cold plate comprising a thermal conduction surface havinga first surface area, and wherein the at least one electronic componentcomprises a surface to be cooled, the surface to be cooled comprising asecond surface area, wherein the liquid-cooled cold plate is over-sizedrelative to the at least one electronic component, with the firstsurface area of the thermal conduction surface being greater than thesecond surface area of the surface to be cooled; and a thermal spreaderin association with the liquid-cooled cold plate, the thermal spreadercomprising at least one heat pipe having multiple heat pipe sections andhaving opposing first and second main surfaces, the first main surfaceof the thermal spreader being coupled to the thermal conduction surfaceof the liquid-cooled cold plate. The method further includes couplingthe liquid-cooled cold plate with the associated thermal spreader to thesurface to be cooled of the at least one electronic component, whereinthe thermal spreader is disposed between the at least one electroniccomponent and the liquid-cooled cold plate, and the liquid-cooled coldplate comprises a first region where the surface to be cooled aligns tothe liquid-cooled cold plate and a second region outside of the firstregion, and wherein at least one coolant-carrying channel section of theplurality of coolant-carrying channel sections is disposed, at least inpart, within the first region, and multiple coolant-carrying channelsections of the plurality of coolant-carrying channel sections of theliquid-cooled cold plate are disposed in the second region, outside thefirst region, and at least one heat pipe section of the multiple heatpipe sections is partially aligned to the first region of theliquid-cooled cold plate, between the at least one electronic componentand the liquid-cooled cold plate, and partially aligned to the secondregion of the liquid-cooled cold plate, away from the at least oneelectronic component, the at least one heat pipe of the thermal spreaderfacilitating distribution of heat outward from the at least oneelectronic component to the multiple coolant-carrying channel sectionsof the liquid-cooled cold plate in the second region of theliquid-cooled cold plate.

In another aspect, a method is provided which includes: providing atleast one electronic component; and providing a cooling apparatuscoupled to the at least one electronic component for dissipating heatfrom the at least one electronic component. The cooling apparatus has alarger footprint size than the at least one electronic component, andincludes: a liquid-cooled cold plate comprising a thermally conductivematerial with a plurality of coolant-carrying channel sections extendingtherein, the liquid-cooled cold plate comprising a thermal conductionsurface having a first surface area, and wherein the at least oneelectronic component comprises a surface to be cooled, the surface to becooled comprising a second surface area, wherein the liquid-cooled coldplate is over-sized relative to the at least one electronic component,with the first surface area of the thermal conduction surface of theliquid-cooled cold plate being greater than the second surface area ofthe surface to be cooled of the at least one electronic component, andin operation, heat is transferred from the surface to be cooled of theat least one electronic component to the thermal conduction surface ofthe liquid-cooled cold plate, and the liquid-cooled cold plate comprisesa first region where the surface to be cooled aligns to theliquid-cooled cold plate and a second region outside the first region,and wherein at least one coolant-carrying channel section of theplurality of coolant-carrying channel sections is disposed, at least inpart, within the first region, and multiple coolant-carrying channelsections of the plurality of coolant-carrying channel sections of theliquid-cooled cold plate are disposed in the second region, outside thefirst region; and a thermal spreader disposed between the at least oneelectronic component and the liquid-cooled cold plate and havingopposing first and second main surfaces, the first main surface beingcoupled to the thermal conduction surface of the liquid-cooled coldplate, the thermal spreader comprising at least one heat pipe havingmultiple heat pipe sections, at least one heat pipe section of themultiple heat pipe sections being partially aligned to the first regionof the liquid-cooled cold plate, between the at least one electroniccomponent and the liquid-cooled cold plate, and partially aligned to thesecond region of the liquid-cooled cold plate, away from the at leastone electronic component, the at least one heat pipe of the thermalspreader facilitating distribution of heat outward from the at least oneelectronic component to the multiple coolant-carrying channel sectionsof the liquid-cooled cold plate in the second region of theliquid-cooled cold plate.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 is a cross-sectional plan view of one embodiment of anelectronics rack with an attached air-to-liquid heat exchanger enhancingcooling of air passing through the electronics rack;

FIG. 3 depicts one embodiment of a data center with a coolantdistribution unit facilitating liquid-cooling of one or moreliquid-cooled electronics racks of the data center, in accordance withan aspect of the present invention;

FIG. 4 depicts an alternate embodiment of a cooling apparatus andliquid-cooled electronics rack, in accordance with one or more aspectsof the present invention;

FIG. 5A is a more detailed, elevational view of one embodiment of theliquid-cooled electronics rack of FIG. 4, and illustrating rack-levelcoolant distribution structures, in accordance with one or more aspectsof the present invention;

FIG. 5B is a partial depiction of a more detailed embodiment of therack-level coolant distribution structures illustrated in FIG. 5A, inaccordance with one or more aspects of the present invention;

FIG. 6 is a plan view of one embodiment of an electronic system layoutfor a liquid-cooled electronics rack, and illustrating multipleliquid-cooled cold plates and multiple liquid-cooled cold rails coupledin fluid communication, in accordance with one or more aspects of thepresent invention;

FIG. 7 depicts one detailed embodiment of a partially assembledelectronic system, wherein the electronic system includes multipledifferent types of heat-generating electronic devices to be cooled, inaccordance with one or more aspects of the present invention;

FIG. 8A depicts the electronic system of FIG. 7, with first, second andthird liquid-cooled cold rails of a cooling apparatus shown in place atthe ends of first and second arrays of sockets and electronics cards ofthe electronic system, in accordance with one or more aspects of thepresent invention;

FIG. 8B is a partial depiction of the partially assembled cooledelectronic system of FIG. 8A, and illustrating in greater detail, oneembodiment of the first liquid-cooled cold rail disposed at one end ofthe parallel-disposed sockets that form part of the first array ofelectronics cards, in accordance with one or more aspects of the presentinvention;

FIG. 8C is a partial depiction of the second liquid-cooled cold raildisposed at the other end of the sockets comprising part of the firstarray of electronics cards, and shown disposed between the first andsecond arrays of electronics cards illustrated in FIG. 8A, in accordancewith one or more aspects of the present invention;

FIG. 8D depicts one embodiment of the third liquid-cooled cold raildisposed at the other end of the sockets that form part of the secondarray of electronics cards illustrated in FIG. 8A, in accordance withone or more aspects of the present invention;

FIG. 9A depicts the partially assembled, cooled electronic system ofFIGS. 8A-8D, with a plurality of thermal spreaders shown coupled to theelectronics cards and thermally interfacing the electronics cards torespective liquid-cooled cold rails, in accordance with one or moreaspects of the present invention;

FIG. 9B is a partial depiction of the cooled electronic system of FIG.9A, and illustrating interfacing of thermal spreaders to the firstliquid-cooled cold rail at the one end of the sockets of the first arrayof electronics cards, in accordance with one or more aspects of thepresent invention;

FIG. 9C is a partial depiction of the cooled electronic system of FIG.9A, and illustrating the second liquid-cooled cold rail disposed betweenthe first and second arrays of electronics cards, and showinginterfacing of thermal spreaders coupled to the first array ofelectronics cards, and thermal spreaders coupled to the second array ofelectronics cards to the second liquid-cooled cold rail, in accordancewith one or more aspects of the present invention;

FIG. 9D is a partial depiction of the cooled electronic system of FIG.9A, and illustrating interfacing of thermal spreaders associated withthe second array of electronics cards to the third liquid-cooled coldrail of the cooled electronic system, in accordance with one or moreaspects of the present invention;

FIG. 10A is a cross-sectional view of one embodiment of a coolingapparatus comprising a liquid-cooled cold plate with a plurality ofcoolant-carrying micro-channels formed therein, in accordance with oneor more aspects of the present invention;

FIG. 10B is a plan view of an alternate embodiment of a coolingapparatus comprising a liquid-cooled cold plate, shown overlying anelectronic component to be cooled, wherein the liquid-cooled cold platecomprises a plurality of coolant-carrying tubes embedded within thethermally conductive material of the cold plate, in accordance with oneor more aspects of the present invention;

FIG. 11A depicts an alternate embodiment of a cooling apparatuscomprising a liquid-cooled cold plate including one or morecoolant-carrying tubes embedded within the thermally conductive materialof the cold plate, and shown coupled to an electronic component forcooling the electronic component, in accordance with one or more aspectsof the present invention;

FIG. 11B is an elevational view of the liquid-cooled cold plate andelectronic component of FIG. 11A, in accordance with one or more aspectsof the present invention;

FIG. 12A depicts one embodiment of a cooling apparatus comprising aliquid-cooled cold plate and thermal spreader embedded within a common,thermally conductive structure, and shown coupled to an electroniccomponent to be cooled, in accordance with one or more aspects of thepresent invention;

FIG. 12B depicts an alternate embodiment of a cooling apparatuscomprising a liquid-cooled cold plate and a thermal spreader, with thecooling apparatus shown coupled to an electronic component to be cooled,in accordance with one or more aspects of the present invention;

FIG. 12C is an elevational view of the cooling apparatus of FIG. 12B, inaccordance with one or more aspects of the present invention;

FIG. 13 is a graph of thermal modeling results for a cooling apparatuscomprising a liquid-cooled cold plate without a thermal spreader incomparison to a cooling apparatus comprising a liquid-cooled cold platewith an associated thermal spreader such as described herein, inaccordance with one or more aspects of the present invention;

FIG. 14A depicts an alternate embodiment of a cooling apparatuscomprising a liquid-cooled cold plate coupled to an electronic componentin a manner offset from the center of the liquid-cooled cold plate, inaccordance with one or more aspects of the present invention;

FIG. 14B depicts the cooling apparatus of FIGS. 12A-12C, coupled to anelectronic component in a manner offset from the center of theliquid-cooled cold plate, in accordance with one or more aspects of thepresent invention;

FIG. 15A is an elevational view of an alternate embodiment of a coolingapparatus comprising a liquid-cooled cold plate and a thermal spreader,in accordance with one or more aspects of the present invention;

FIG. 15B depicts one embodiment of a thermal spreader for the coolingapparatus of FIG. 15A, in accordance with one or more aspects of thepresent invention;

FIG. 15C depicts one embodiment of a liquid-cooled cold plate for thecooling apparatus of FIG. 15A, in accordance with one or more aspects ofthe present invention;

FIG. 16A depicts an alternate embodiment of a cooling apparatuscomprising a liquid-cooled cold plate and a thermal spreader, and showncoupled to an electronic component to be cooled, in accordance with oneor more aspects of the present invention;

FIG. 16B depicts another embodiment of a cooling apparatus comprising aliquid-cooled cold plate and multiple thermal spreaders, and showncoupled to an electronic component to be cooled, in accordance with oneor more aspects of the present invention;

FIG. 16C illustrates the cooling apparatus of FIG. 16A, shown coupled tomultiple electronic components to be cooled, in accordance with one ormore aspects of the present invention;

FIG. 17A depicts an alternate embodiment of a thermal spreader for acooling apparatus such as disclosed herein, and shown coupled to anelectronic component to be cooled, in accordance with one or moreaspects of the present invention;

FIG. 17B depicts another embodiment of a thermal spreader for a coolingapparatus such as disclosed herein, and shown coupled to an electroniccomponent to be cooled, in accordance with one or more aspects of thepresent invention;

FIG. 17C depicts an alternate embodiment of a thermal spreader for acooling apparatus such as disclosed herein, and shown coupled to anelectronic component to be cooled, in accordance with one or moreaspects of the present invention; and

FIG. 17D depicts another embodiment of a thermal spreader for a coolingapparatus such as disclosed herein, and shown coupled to an electroniccomponent to be cooled, in accordance with one or more aspects of thepresent invention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, “rack-mounted electronicequipment”, and “rack unit” are used interchangeably, and unlessotherwise specified include any housing, frame, rack, compartment, bladeserver system, etc., having one or more heat generating components of acomputer system or electronics system, and may be, for example, astand-alone computer processor having high, mid or low end processingcapability. In one embodiment, an electronics rack may comprise aportion of an electronic system, a single electronic system or multipleelectronic systems, for example, in one or more sub-housings, blades,books, drawers, nodes, compartments, etc., having one or moreheat-generating electronic components disposed therein. An electronicsystem(s) within an electronics rack may be movable or fixed relative tothe electronics rack, with rack-mounted electronic drawers and blades ofa blade center system being two examples of electronic systems (orsubsystems) of an electronics rack to be cooled.

“Electronic component” refers to any heat-generating electroniccomponent of, for example, a computer system or other electronic systemrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies, and/or other electronicdevices to be cooled, such as one or more electronics cards comprising aplurality of memory modules (such as one or more dual in-line memorymodules (DIMMs)).

Further, as used herein, the terms “liquid-cooled structure”,“liquid-cooled cold plate” and “liquid-cooled cold rail” refer tothermally conductive structures having one or more channels (orpassageways) formed therein or passing therethrough, which facilitatethe flow of liquid coolant through the structure. A liquid-cooledstructure may be, for example, a liquid-cooled cold plate or aliquid-cooled cold rail. In one example, tubing is provided extendingthrough the liquid-cooled structure. An “air-to-liquid heat exchanger”or “air-to-liquid heat exchange assembly” means any heat exchangemechanism characterized as described herein through which liquid coolantcan circulate; and includes, one or more discrete air-to-liquid heatexchangers coupled either in series or in parallel. An air-to-liquidheat exchanger may comprise, for example, one or more coolant flowpaths, formed of thermally conductive tubing (such as copper or othertubing) in thermal or mechanical contact with a plurality of air-cooledcooling fins. Size, configuration and construction of the air-to-liquidheat exchanger can vary without departing from the scope of theinvention disclosed. Still further, “data center” refers to a computerinstallation containing one or more electronics racks to be cooled. As aspecific example, a data center may comprise one or more rows ofrack-mounted computer units, such as server units.

One example of coolant used within the cooled electronic apparatusesdisclosed herein is water. However, the concepts presented are readilyadapted to use with other types of coolant. For example, the coolant maycomprise a brine, a fluorocarbon liquid, a liquid metal, or othersimilar coolant, or refrigerant, while still maintaining the advantagesand unique features of the present invention.

Reference is made below to the drawings, which are not drawn to scalefor reasons of understanding, wherein the same reference numbers usedthroughout different figures designate the same or similar components.

FIG. 1 depicts a raised floor layout of an air cooled data center 100typical in the prior art, wherein multiple electronics racks 110 aredisposed in one or more rows. A data center such as depicted in FIG. 1may house several hundred, or even several thousand microprocessors. Inthe arrangement illustrated, chilled air enters the computer room viaperforated floor tiles 160 from a supply air plenum 145 defined betweenthe raised floor 140 and a base or sub-floor 165 of the room. Cooled airis taken in through louvered covers at air inlet sides 120 of theelectronics racks and expelled through the back (i.e., air outlet sides130) of the electronics racks. Each electronics rack 110 may have one ormore air moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet airflow to cool the electronic devices within the rackunit. The supply air plenum 145 provides conditioned and cooled air tothe air-inlet sides of the electronics racks via perforated floor tiles160 disposed in a “cold” aisle of the computer installation. Theconditioned and cooled air is supplied to plenum 145 by one or more airconditioning units 150, also disposed within the data center 100. Roomair is taken into each air conditioning unit 150 near an upper portionthereof. This room air may comprise, in part, exhausted air from the“hot” aisles of the computer installation defined, for example, byopposing air outlet sides 130 of the electronics racks 110.

Due to ever-increasing air flow requirements through electronics racks,and the limits of air distribution within a typical data centerinstallation, liquid-based cooling is being combined with conventionalair-cooling. FIGS. 2-4 illustrate various embodiments of a data centerimplementation employing a liquid-based cooling system.

FIG. 2 depicts one rack-level liquid-cooling solution which utilizeschilled facility water to remove heat from the computer installationroom, thereby transferring the cooling burden from the air-conditioningunit(s) to the building's chilled water coolers. The embodiment depictedin FIG. 2 is described in detail in commonly assigned, U.S. Pat. No.6,775,137. Briefly summarized, facility-chilled water 200 circulatesthrough one or more liquid-to-liquid heat exchangers 210, coupled via asystem coolant loop 211, to individual electronics racks 220 within thecomputer room. Rack unit 220 includes one or more air-moving devices 230for moving air flow from an air inlet side to an air outlet side acrossone or more drawer units 240 containing heat-generating electroniccomponents to be cooled. In this embodiment, a front cover 250 attachedto the rack covers the air inlet side, a back cover 255 attached to therack covers the air outlet side, and a side car disposed adjacent to(and/or attached to) the rack includes a heat exchanger 260 for coolingair circulating through the rack unit. Further, in this embodiment, theliquid-to-liquid heat exchangers 210 are multiple computer roomwater-conditioning (CRWC) units which are coupled to receive buildingchilled facility water 200. The building chilled facility water is usedto cool the system coolant within system coolant loop 211, which iscirculating through air-to-liquid heat exchanger 260. The rack unit inthis example is assumed to comprise a substantially enclosed housing,wherein the same air circulates through the housing that passes acrossthe air-to-liquid heat exchanger 260. In this manner, heat generatedwithin the electronics rack is removed from the enclosed housing via thesystem coolant loop, and transferred to the facility coolant loop forremoval from the computer installation room.

FIG. 3 depicts another embodiment of a rack-level, liquid-coolingsolution, which again uses chilled facility water to remove heat fromthe computer installation room, thereby transferring the cooling burdenfrom the air-conditioning unit(s) to the building's chilled watercoolers. In this implementation, one embodiment of a coolantdistribution unit 300 for a data center is illustrated. Within coolantdistribution unit 300 is a power/control element 312, areservoir/expansion tank 313, a liquid-to-liquid heat exchanger 314, apump 315 (often accompanied by a redundant second pump), facility waterinlet 316 and outlet 317 supply pipes, a supply manifold 318 supplyingwater or system coolant to the electronics racks 110 via couplings 320and lines 322, and a return manifold 319 receiving water or systemcoolant from the electronics racks 110, via lines 323 and couplings 321.Each electronics rack includes (in one example) a power/control unit 330for the electronics rack, multiple electronic systems or subsystems 340,a system coolant supply manifold 350, and a system coolant returnmanifold 360. As shown, each electronics rack 110 is disposed on raisedfloor 140 of the data center with lines 322 providing system coolant tosystem coolant supply manifolds 350 and lines 323 facilitating return ofsystem coolant from system coolant return manifolds 360 being disposedin the supply air plenum beneath the raised floor.

In the embodiment illustrated, system coolant supply manifold 350provides system coolant to cooling apparatuses disposed within theelectronic systems or subsystems (for example, to liquid-cooled coldplates or cold rails) via flexible hose connections 351, which aredisposed between the supply manifold and the respective electronicsystems within the rack. Similarly, system coolant return manifold 360is coupled to the electronic systems via flexible hose connections 361.Quick connect couplings may be employed at the interface betweenflexible hoses 351, 361 and the individual electronic systems. By way ofexample, these quick connect couplings may comprise various types ofcommercially available couplings, such as those available from ColderProducts Company, of St. Paul, Minn., USA, or Parker Hannifin, ofCleveland, Ohio, USA.

Although not shown, electronics rack 110 may also include anair-to-liquid heat exchanger, for example, disposed at an air outletside thereof, which also receives system coolant from the system coolantsupply manifold 350 and returns system coolant to the system coolantreturn manifold 360.

FIG. 4 illustrates another embodiment of a liquid-cooled electronicsrack and cooling system therefor, in accordance with one or more aspectsof the present invention. In this embodiment, the electronics rack 400has a side car structure 410 associated therewith or attached thereto,which includes an air-to-liquid heat exchanger 415 through which aircirculates from an air outlet side of electronics rack 400 towards anair inlet side of electronics rack 400, for example, in a closed looppath in a manner similar to that illustrated above in connection withthe cooling implementation of FIG. 2. In this example, the coolingsystem comprises an economizer-based, warm-liquid coolant loop 420,which comprises multiple coolant tubes (or lines) connecting, in theexample depicted, air-to-liquid heat exchanger 415 in series fluidcommunication with a coolant supply manifold 430 associated withelectronics rack 400, and connecting in series fluid communication, acoolant return manifold 431 associated with electronics rack 400, acooling unit 440 of the cooling system, and air-to-liquid heat exchanger415.

As illustrated, coolant flowing through warm-liquid coolant loop 420,after circulating through air-to-liquid heat exchanger 415, flows viacoolant supply plenum 430 to one or more electronic systems ofelectronics rack 400, and in particular, one or more cold plates and/orcold rails 435 associated with the electronic systems, before returningvia coolant return manifold 431 to warm-liquid coolant loop 420, andsubsequently to a cooling unit 440 disposed (for example) outdoors fromthe data center. In the embodiment illustrated, cooling unit 440includes a filter 441 for filtering the circulating liquid coolant, acondenser (or air-to-liquid heat exchanger) 442 for removing heat fromthe liquid coolant, and a pump 443 for returning the liquid coolantthrough warm-liquid coolant loop 420 to air-to-liquid heat exchanger415, and subsequently to the liquid-cooled electronics rack 400. By wayof example, hose barb fittings 450 and quick disconnect couplings 455may be employed to facilitate assembly or disassembly of warm-liquidcoolant loop 420.

In one example of the warm coolant-cooling approach of FIG. 4, ambienttemperature might be 30° C., and coolant temperature 35° C. leaving theair-to-liquid heat exchanger 442 of the cooling unit. The cooledelectronic system depicted thus facilitates a chiller-less data center.Advantageously, such a liquid-cooling solution provides highly energyefficient cooling of the electronic systems of the electronics rack,using liquid (e.g., water), that is cooled via circulation through theair-to-liquid heat exchanger located outdoors (i.e., a dry cooler) withexternal ambient air being pumped through the dry cooler. Note that thiswarm coolant-cooling approach of FIG. 4 is presented by way of exampleonly. In alternate approaches, cold coolant-cooling could be substitutedfor the cooling unit 440 depicted in FIG. 4. Such cold coolant-coolingmight employ building chilled facility coolant to cool the liquidcoolant flowing through the liquid-cooled electronics rack, andassociated air-to-liquid heat exchanger (if present), in a manner suchas described above in connection with FIGS. 2 & 3.

FIGS. 5A & 5B depict in greater detail one embodiment of a liquid-cooledelectronics rack, such as depicted in FIG. 4, in accordance with one ormore aspects of the present invention. In this implementation,liquid-cooled electronics rack 400 comprises a plurality of electronicsystems 500, within which one or more electronic components are to beliquid-cooled via, for example, one or more cold plates or cold rails,as described below. The cooling system includes coolant loop 420 coupledin fluid communication with coolant supply manifold 430 and coolantreturn manifold 431, both of which may comprise vertically-orientedmanifolds attached to liquid-cooled electronics rack 400. In thisembodiment, the rack-level coolant distribution system further includesindividual node-level supply hoses 510 supplying coolant from coolantsupply manifold 430 to cold plates and cold rails within the electronicsystems 500. As shown in FIG. 5B, coolant supply manifold 430 may be (inone embodiment) a vertically-oriented manifold with a plurality ofcoupling connections 511 disposed along the manifold, one for eachelectronic system 500 having one or more electronic components to beliquid-cooled. Coolant leaves the individual electronic systems 500 vianode-level return hoses 520, which couple the individual electronicsystems (or nodes) to coolant return manifold 431, and hence, to coolantloop 420. In the embodiment illustrated in FIG. 4, relativelywarm-liquid coolant, such as water, is supplied from the cooling unit,either directly, or through one or more air-to-liquid heat exchanger(s)415 (of FIG. 4), and hot coolant is returned via the coolant returnmanifold to the cooling unit. In one embodiment of the rack-levelcoolant distribution system illustrated in FIGS. 5A & 5B, the node-levelsupply and return hoses 510, 520 are flexible hoses.

FIG. 6 illustrates one embodiment of a cooled electronic system 500component layout, wherein one or more air-moving devices 600 provideforced air flow 601 to cool multiple components 610 within electronicsystem 500. Cool air is taken in through a front 602 and exhausted out aback 603 of the electronic system (or drawer). The multiple componentsto be cooled include, for example, multiple processor modules to whichliquid-cooled cold plates 620 (of the liquid-based cooling apparatus)are coupled, as well as multiple arrays 631, 632 of electronics cards630 (e.g., memory modules such as dual in-line memory modules (DIMMs)),which are to be thermally coupled to one or more liquid-cooled coldrails 625. As used herein “thermally coupled” refers to a physicalthermal transport path being established between components, forexample, between an electronics card and a liquid-cooled cold rail forthe conduction of heat from one to the other.

The illustrated liquid-based cooling approach further includes multiplecoolant-carrying tubes connecting in fluid communication liquid-cooledcold plates 620 and liquid-cooled cold rails 625. These coolant-carryingtubes comprise (for example), a coolant supply tube 640, multiple bridgetubes 641, and a coolant return tube 642. In the embodiment illustrated,bridge tubes 641 connect one liquid-cooled cold rail 625 in seriesbetween the two liquid-cooled cold plates 620, and connect in paralleltwo additional liquid-cooled cold rails 625 between the secondliquid-cooled cold plate 620 and the coolant return tube 642. Note thatthis configuration is provided by way of example only. The conceptsdisclosed herein may be readily adapted to use with variousconfigurations of cooled electronic system layouts. Note also, that asdepicted herein, the liquid-cooled cold rails are elongate, thermallyconductive structures comprising one or more channels through whichliquid coolant passes, for example, via one or more tubes extendingthrough the structures. The liquid-cooled cold rails are disposed, inthe embodiment illustrated, at the ends of the two arrays (or banks)631, 632 of electronics cards 630, and multiple thermal spreaders areprovided coupling in thermal communication electronics cards 630 andliquid-cooled cold rails 625. Various such thermal spreaders arediscussed below with reference to FIGS. 8A-19C.

FIG. 7 depicts in greater detail one embodiment of an electronic systemlayout comprising a printed circuit board 700 with two processor modules710, each of which is to have a respective liquid-cooled cold plate of aliquid-based cooling system coupled thereto, and multiple arrays 721,722 of electronics cards 720, such as memory cards comprising memorymodules on opposite first and second sides thereof. Electronics cards720 are held in respective sockets 730, mounted to printed circuit board700, and latches 740 at the ends of sockets 730 facilitate securing (orremoving) of electronics cards 720 within (or from) the respectivesockets 730. The cooling apparatus embodiments described hereinbelowadvantageously facilitate liquid-cooling of electronics cards 720without interfering with an operator's access to latches 740 at the endsof sockets 730. In addition to existing component constraints on thesurface of printed circuit board 700, there is assumed to be negligiblespace between a cover (not shown) of the electronic system (e.g.,server), and the top edge surfaces 725 of electronics cards 720.

FIGS. 8A-8D depicts a partial assembly of a cooled electronic systemcomprising the electronic system layout of FIG. 7, including printedcircuit board 700, processor modules 710, and arrays 721, 722 ofelectronics cards 720. Electronics cards 720 are each shown positionedwithin a respective socket 730 mounted to printed circuit board 700,with latches 740 being disposed at the opposite ends of each socket 730.Latches 740 facilitate securing (or removing) electronics cards 720within (or from) the sockets.

FIGS. 8A-8D further depict multiple liquid-cooled cold rails 800, 810,820, shown positioned at the ends of the elongate sockets 730 of the twoarrays 721, 722 of electronics cards 720. Advantageously, theseliquid-cooled cold rails are configured and positioned to not interferewith opening and closing of latches 740. The multiple cold rails includea first liquid-cooled cold rail 800, disposed at one end of sockets 730in the first array 721 of electronics cards 720, a second liquid-cooledcold rail 810 disposed between the two arrays 721, 722 of electronicscards 720, and a third liquid-cooled cold rail 820 disposed at the otherend of sockets 730 of the second array 722 of electronics cards 720.Holes 830 are provided within each of the cold rails. In one embodiment,these holes may comprise threaded holes in the cold rails whichfacilitate attachment of the thermal spreaders (not shown) to therespective cold rails, as described further below.

In FIG. 8B, first liquid-cooled cold rail 800 is illustrated in greaterdetail at the one end of the sockets 730 of the first array 721 ofelectronics cards 720. As noted above, each cold rail is a thermallyconductive structure with at least one coolant-carrying channelextending therein. In this example, the coolant-carrying channel is aflattened tube 801 that is vertically-oriented within the cold rail andoffset from holes 830. As shown, first liquid-cooled cold rail 800 issized (in this example) to fit between sockets 730, and one or more fansockets 805. In addition, the cold rail may be selectively recessed atits bottom surface and/or one or more side surfaces to clear anyinterfering components, such as capacitors or chips, on the printedcircuit board. The flattened tube 801 extending through the thermallyconductive structure of the cold rail may comprise (as one example) aflattened ⅛ inch pipe, which may be routed above any intrusive elementson the board. In the example illustrated, quick disconnects 802 (FIG.8A) are shown provided at the ends of flattened tube 801.

In FIG. 8C, second liquid-cooled cold rail 810 is illustrated in greaterdetail. This cold rail is configured and sized to fit between the twoarrays 721, 722 of electronics cards. Two rows of holes 830 are providedwithin second liquid-cooled cold rail 810 to facilitate coupling ofthermal spreaders from the different arrays to the cold rail. In orderto clear the holes in the cold rail, a vertically-oriented, flattenedpipe 811 passing through cold rail 810 is positioned within a slot 813cut in the middle of the cold rail, for example, from the lower surfaceof the cold rail into the thermally conductive structure. By way ofexample, flattened tube 811 may be a flattened ⅛ inch pipe. Quickdisconnect couplings 802 may also be provided for connecting flattenedtube 811 in fluid communication with other coolant-carrying tubes, suchas the bridging tubes described above in connection with FIG. 6.

FIG. 8D illustrates in greater detail one embodiment of thirdliquid-cooled cold rail 820 disposed at the other end of the secondarray 722 of electronics cards 720. As illustrated, third liquid-cooledcold rail 820 is positioned to not interfere with opening or closing oflatches 740 at the other end of sockets 720 in the second array 722. Thecold rail includes a series of holes 830, which will facilitate couplingthermal spreaders (not shown) to the cold rail, and accommodates aflattened tube 821, which is vertically aligned within anappropriately-sized slot 823 in the thermally conductive structure ofthe cold rail and is offset from the series of holes 830. This cold railmay again be selectively recessed at its lower surface and/or sidesurfaces to clear any interfering components on printed circuit board700.

In the example of FIG. 8D, and assuming the cooling implementationdepicted in FIG. 6, the tube through which water flows may be aflattened, ¼ inch pipe, routed away from any intrusive elements on theprinted circuit board. As illustrated in FIG. 8A, quick disconnectcouplings 802 may be provided at the ends of flattened tube 820 tofacilitate coupling of the tube in fluid communication with other tubesof the liquid-based cooling approach discussed above in connection withFIG. 6. As illustrated in FIGS. 8A-8D, each liquid-cooled cold rail 800,810, 820 may be unique in terms of its location on the circuit board,and uniquely configured due to existing constraints within the differentareas of the printed circuit board. These liquid-cooled cold rails are,in one embodiment, coupled to either a cold liquid cooling loop or awarm-liquid cooling loop, depending on the cooling approach desired, asdescribed above.

FIGS. 9A-9D depict one embodiment of a cooled electronic systemcomprising the electronic subassembly of FIGS. 8A-8D, with a pluralityof thermal spreaders shown positioned between and in physical andthermal contact with the electronics cards of the arrays (or banks) ofelectronics cards. These thermal spreaders provide a thermal coupling orthermal conduction path from the electronics cards, for example, thememory modules on the opposite sides of the electronics cards, to theliquid-cooled cold rails to facilitate cooling of the electronics cardsvia conductive heat transfer to the cold rails, and hence to the liquidflowing through the cold rails.

In the embodiment illustrated, each thermal spreader comprises a firstthermal transfer plate 910 and a second thermal transfer plate 920. Thefirst thermal transfer plate comprises a first thermal conductionsurface, and the second thermal transfer plate 920 comprises a secondthermal conduction surface. The first thermal conduction surface and thesecond thermal conduction surface are in spaced, opposing relation, andare configured to accommodate a respective electronics card 720therebetween, with the first thermal conduction surface physically andthermally coupled to at least one first surface on one side of theelectronics card 720, and the second thermal conduction surfacephysically and thermally coupled to at least one second surface on theother side of the electronics card 720. These first and second surfaceson the different sides of the electronics card may comprise, in oneexample, surfaces of one or more electronics devices, such as memorymodules, mounted on the different sides of the respective electronicscard.

Further, the first thermal transfer plate 910 and second thermaltransfer plate 920 each comprise a first end edge 915, and a second endedge 916, disposed at opposite ends of the respective socket 730. Eachthermal transfer plate is a thermally conductive structure formed (inone example) as an elongate, flat plate. In this example, thermallyconductive extensions 912, 922 and 913, 923 are provided extending fromthe first and second end edges 915, 916 of each thermal transfer plate910, 920.

In one embodiment, these extensions 912, 922 and 913, 923 are curvedextensions, which may be characterized, in one embodiment, as “elephanttrunk-shaped extensions”. In particular, a first thermally conductiveextension 912 is a curved extension which extends from and upper portionof first thermal transfer plate 910 at the first end edge thereof 915,and a similar, second thermally conductive extension 922 extends fromthe first end edge 915 of second thermal transfer plate 920. Inaddition, a third thermally conductive extension 913 extends from thesecond end edge 916 of first thermal transfer plate 910, and a fourththermally conductive extension 923 extends from the second end edge 916of second thermal transfer plate 920. The thermally conductiveextensions 912, 922 at the first end edge 915 of the first and secondthermal transfer plates 910, 920 are spaced apart to allow access to therespective socket latch at the end of the socket 730 containing theelectronics card 720 sandwiched by the plates of the thermal spreader.Similarly, the thermally conductive extensions 913, 923 at the secondend edges 916 of the first and second thermal transfer plates 910, 920are spaced apart to allow access to the latch disposed at the other endof the socket. In this embodiment, the extensions 912, 922 and 913, 923are joined at their ends, and connected to the respective cold rail byrespective connecting flanges 930, each of which includes an opening935, aligned to an underlying opening 830 in the adjacent cold rail 800,810, 820. FIGS. 9B-9D illustrate these structures in greater detail,with the thermal spreaders shown ready to be fastened to the respectivecold rails using, for example, a threaded fastener.

As explained above, heat is transferred from the heat-generatingcomponents of the electronics card (for example, memory modules) to theconduction surfaces of the thermal transfer plates, across the thermaltransfer plates to the thermally conductive extensions at the endsthereof, and from the thermally conductive extensions into therespective liquid-cooled cold rails. From the liquid-cooled cold rails,the heat is rejected to coolant flowing through the channels or tubesextending through the cold rails, and subsequently, is removed from thecooled electronic system in a manner such as, for example, explainedabove in connection with FIGS. 4-6.

Note that in the embodiment depicted in FIGS. 9A-9D, the connectingflanges 930 at the ends of the thermally conductive extensions (wherecontacting the respective cold rails), are solid connecting structures,meaning that the thermal spreaders are (in one embodiment) single-piecestructures. Also, note that, in the approach depicted, heat is conductedby the thermal transfer plates to each end edge of the plates, and inparticular, to the thermally conductive extensions extending from therespective end edges to the two cold rails disposed at the opposite endsof the respective sockets in an array (or bank) of electronics cards.These thermally conductive extensions and connecting flanges physicallyand thermally couple to the upper surface of the respective cold rails.As illustrated in the plan views of FIGS. 9B-9D, the latches for therespective sockets remain accessible by appropriately spacing each pairof thermally conductive extensions to the first and second sides of thelatches at issue. This can be accomplished, in part, by reducing thethickness of the extensions compared with the thickness of the plates,as shown in the plan views of FIGS. 9B-9D.

FIGS. 10A-17D depict various enhancements to the cooling apparatusapproaches described above, and in particular, various enhancements tocooling apparatuses comprising liquid-cooled structures or cold platesfor cooling one or more heat-generating components (such as one or moreprocessor modules).

FIG. 10A depicts a cross-sectional view of one embodiment of aliquid-cooled cold plate 1000. As shown, liquid-cooled cold plate 1000comprises a thermally conductive structure 1010 with a plurality ofmicro-channels 1020 provided therein, through which coolant flows forextracting heat, for example, from one or more heat-generatingcomponents (not shown) coupled to the cold plate. Coolant is receivedinto the micro-channels via a coolant inlet port 1022 and exhausted viaa coolant outlet port 1024, both of which are in fluid communicationwith micro-channels 1020. Providing a cold plate with a plurality ofmicro-channels 1020 facilitates thermal transfer to the coolant flowingthrough the structure, but is difficult and expensive to manufacture dueto constraints associated with forming the micro-channels (which arevery small coolant flow passageways, for example, on the order of 0.5mm). For example, one approach for fabricating the liquid-cooled coldplate 1000 of FIG. 10A with a plurality of micro-channels is to employedskiving/wire EDM/impact extrusion processing, along with vacuum brazing,which can be a difficult and expensive manufacturing approach. Hence,the liquid-cooled structure of FIG. 10A might not be a cost effectivecooling solution. Also, in view of the manufacturing constraints,liquid-cooled cold plate 1000 of FIG. 10A would typically bemanufactured with a similar footprint to the particular surface to becooled of the electronic component, which means that using the coldplate depicted in FIG. 10A, different electronic components to beliquid-cooled would probably require different, specially-configured andsized cold plates.

In FIG. 10B, a different type of liquid-cooled structure or cold plate1050 is depicted, wherein a plurality of coolant-carrying tubes 1070 aredisposed within channels in a thermally conductive structure 1060, whichmight comprise an aluminum or copper block (or plate). By way ofexample, coolant-carrying tubes 1070 may have an outer diameter in therange of 4.5 mm-6.35 mm, and be manufactured by bending tubes to thedesired configuration and mechanically attaching the tubes to channelsin a plate using, for example, thermal epoxy.

In the embodiment illustrated, coolant-carrying tubes 1070 are in fluidcommunication with a coolant inlet plenum 1072 and a coolant outletplenum 1074. Coolant flows via a coolant inlet port 1073 to coolantinlet plenum 1072 for passage through the plurality of coolant-carryingtubes 1070, and heated coolant is exhausted via coolant outlet plenum1074 through a coolant outlet port 1075. In the example of FIG. 10B,liquid-cooled cold plate 1050 is shown to have a substantially largerfootprint than that of the electronic component 1051 to be cooled.Specifically, in the embodiment illustrated, the thermal conductionsurface of the cold plate which couples to the surface to be cooled ofthe electronic component has a larger surface area than that of thesurface to be cooled. Thus, in this implementation, the liquid-cooledcold plate and coolant-carrying tubes may be larger structures than thecorresponding structures of the micro-channel cold plate of FIG. 10A,and therefore the implementation of FIG. 10B is more straightforward andcost effective to manufacture. However, due to its size, thermaltransfer performance to coolant flowing through the liquid-cooled coldplate 1050 of FIG. 10B might be lower than thermal transfer performanceto coolant flowing through a liquid-cooled structure such as depicted inFIG. 10A.

FIGS. 11A & 11B collectively illustrate another embodiment of a coolingapparatus comprising a liquid-cooled structure or cold plate 1100fabricated of a thermally conductive structure 1110 and having one ormore coolant-carrying tubes 1120 extending within or through thethermally conductive structure. In this implementation, a singlecoolant-carrying tube 1020 is illustrated, which is bent into asinusoidal shape within the thermally conductive structure 1110.Coolant-carrying tube 1120 is shown to comprise a plurality ofcoolant-carrying tube (or channel) sections 1130, which (in theembodiment of FIG. 11A) are disposed substantially parallel to andspaced from each other within the thermally conductive structure 1110.Liquid-cooled cold plate 1100 is shown coupled to an electroniccomponent 1140 to be cooled. In particular, a thermal conduction surface1112 of liquid-cooled cold plate 1100 is coupled to a surface 1141 to becooled of electronic component 1140, meaning that the surface to becooled of the electronic component is in opposing relation to a portionof thermal conduction surface 1112 of liquid-cooled cold plate 1110.Thermal conduction from the electronic component to the cold plate canbe facilitated by providing a thermal grease, thermally conductive pad,or other interstitial thermally conductive layer between the thermalconduction surface 1112 and the surface to be cooled 1141. Projectinginto the cold plate where the surface 1141 to be cooled aligns to thecold plate defines a first region 1114 of the cold plate and a secondregion 1116 of the cold plate, which is that region of the cold plateoutside the first region. In first region 1114, heat is directlytransferred via conduction upwards (in this example), from the surface1141 to be cooled into the cold plate 1100. In contrast, to reach thesecond region, heat must radiate outwards laterally once in thethermally conductive material of the cold plate in order to reachcoolant-carrying tube sections 1130 in the second region 1116.Therefore, in this implementation, coolant-carrying tube sections 1130in second region 1116 of liquid-cooled cold plate 1100 individuallyprovide reduced heat transport to coolant compared with acoolant-carrying tube section 1130 within first region 1114 of theliquid-cooled cold plate 1100.

Addressing this issue, presented herein are various cooling apparatuseswhich comprise liquid-cooled cold plates (such as described above inconnection with FIGS. 11A & 11B) with associated thermal spreaderscomprising embedded heat pipes (or vapor chambers). These coolingapparatuses are particularly beneficial in combination with theliquid-cooling approaches described herein, such as thewarm-liquid-cooling approach described above with reference to FIGS.4-9D.

Generally stated, the cooling apparatuses disclosed herein facilitatedissipating heat from an electronic component of an electronic system.In one example, the electronic component comprises an integratedcircuit, such as a processor. The cooling apparatus includes aliquid-cooled cold plate, which comprises a thermally conductivematerial having a plurality of coolant-carrying channel sectionsextending therein. The liquid-cooled cold plate includes a thermalconduction surface comprising a first surface area (with the electroniccomponent comprising a surface to be cooled having a second surfacearea), and the first surface area of the thermal conduction surfacebeing greater than the second surface area of the surface to be cooled.In operation, heat is transferred from the surface to be cooled of theelectronic component to the thermal conduction surface of theliquid-cooled cold plate, and the liquid-cooled cold plate includes afirst region where the surface to be cooled aligns to (for example, isdisposed in opposing relation to) the cold plate, and a second regiondisposed outside the first region. Advantageously, a thermal spreader isassociated with the liquid-cooled cold plate, and includes at least oneheat pipe. The at least one heat pipe comprises multiple heat pipesections, with at least one heat pipe section of the multiple heat pipesections being at least partially aligned to the first region of theliquid-cooled cold plate, and partially aligned to the second region ofthe liquid-cooled cold plate. In operation, the heat pipe(s) of thethermal spreader facilitate distribution of heat from the electroniccomponent to coolant-carrying channel sections of the liquid-cooled coldplate disposed, for example, in the second region of the liquid-cooledcold plate, and therefore enhance thermal transfer to coolant flowingthrough the cold plate.

Advantageously, in the cooling apparatuses disclosed herein, theliquid-cooled cold plate includes at least one coolant-carrying tubeembedded within a thermally conductive structure, with the at least onecoolant-carrying tube comprising the plurality of coolant-carryingchannel sections. As a further enhancement, the thermal spreader and theliquid-cooled cold plate may be integrated into a common, thermallyconductive structure (that is, a common, monolithic structure).Alternatively, the thermal spreader may be a discrete structure that isdetachably coupled to the liquid-cooled cold plate, and (in oneembodiment) detachably coupled to the electronic component. Examples ofthese two approaches are illustrated in FIGS. 12A & 12B.

In FIG. 12A, an integrated cooling apparatus 1200 is illustratedcomprising a common, thermally conductive structure 1210 which houses aliquid-cooled cold plate comprising at least one coolant-carrying tube1220 embedded within the common, thermally conductive structure 1210,and a thermal spreader comprising multiple heat pipe sections 1225. Asin the embodiment described above in connection with FIG. 11A, thecoolant-carrying tube is a sinusoidal-shaped tube, which has multiplecoolant-carrying channel sections 1221 disposed in parallel within thecommon, thermally conductive structure 1210. Further, as noted above, aprojection into thermally conductive structure 1210 where electroniccomponent 1230 aligns to the thermally conductive structure, defines afirst region 1214 of thermally conductive structure 1210, and a secondregion 1216 of the thermally conductive structure, which is that regionof the structure outside of the first region 1214. Advantageously,multiple heat pipe sections 1225 are disposed between thecoolant-carrying tube(s) and the electronic component, and partiallyalign to first region 1214 of the thermally conductive structure 1210,and extend into the second region 1216 of the thermally conductivestructure. These multiple heat pipe sections 1225 facilitate lateraldistribution of heat within the common, thermally conductive structure,thereby facilitating conduction of heat from the electronic component tocoolant-carrying channel sections 1221 of the liquid-cooled cold platedisposed in the second region of the thermally conductive structure1210.

FIGS. 12B & 12C depict a substantially identical cooling apparatus 1200′to that described above in connection with FIG. 12A. One difference,however, is that the liquid-cooled cold plate 1240 and thermal spreader1250 are fabricated as discrete components (e.g., plates) which aredetachably coupled to each other, with thermal spreader 1250 residingbetween liquid-cooled cold plate 1240 and electronic component 1230 tobe cooled. Thermal conduction from electronic component 1230 to thermalspreader 1250, and from thermal spreader 1250 to liquid-cooled coldplate 1240, can be facilitated by providing a thermal grease, thermallyconductive pad, or other interstitial thermally conductive layer betweenthe respective opposing surfaces.

As explained, liquid-cooled cold plate 1240 includes one or morecoolant-carrying tubes 1220, which in this example, comprise a singlecoolant-carrying tube 1220 extending in a sinusoidal manner within thethermally conductive structure 1241 of liquid-cooled cold plate 1240.Projecting where electronic component 1230 aligns to the liquid-cooledcold plate defines a first region 1214 and a second region 1216 of thecold plate, wherein one or more coolant-carrying channel sections 1221may be disposed only in the second region of the liquid-cooled coldplate.

Thermal spreader 1250 comprises a thermally conductive structure withinwhich multiple heat pipe sections 1225 are disposed, as described abovein connection with the implementation of FIG. 12A. By way of example,these multiple heat pipe sections 1225 comprise multiple straight heatpipe sections positioned, at least in part, parallel and physicallycontacting each other lengthwise in the region between the electroniccomponent and the liquid-cooled cold plate. These straight heat pipesections 1225 align at least partially to the first region 1214 ofliquid-cooled cold plate and extend into the second region 1216 of thecold plate to facilitate lateral spreading of heat from the electroniccomponent into the second region of the liquid-cooled cold plate, andthereby facilitate thermal transfer to coolant flowing throughcoolant-carrying channel sections 1221 located in second region 1216 ofliquid-cooled cold plate 1240. Note that in an alternate embodiment, theentire thermal spreader could be a wall of straight heat pipe sectionsdisposed parallel and physically contacting along their lengthsthroughout the entire thermal spreader.

Note also that there are a number of advantages to the cooling apparatusdesigns depicted in FIGS. 12A-12C. For example, one problem solved bythese cooling apparatuses is the manufacturability of the cold plates.For a cold plate with a micro-channel design, such as illustrated inFIG. 10A, manufacturing is often cost-prohibitive (for example, becausethe cold plate would typically need to be vacuum-brazed). In contrast,in a cooling apparatus embodiment such as proposed herein employing athermal spreader (with embedded heat pipes), the larger,coolant-carrying tubes can be readily soldered into an appropriatelyconfigured cold plate (with the soldering enhancing heat transfer fromthe cold plate to coolant flowing through the tubes). This approach is amore cost effective manufacturing approach in comparison to themicro-channel approach described above.

Note further that as used herein, a “heat pipe” (or vapor chamber)comprises (for example) a hollow structure that allows heat to migratefrom one end of the structure to another end via a flow of vapor. Theheat pipe provides a return of liquid condensate under the action ofcapillary force via a wick or screen-like matrix. A high concentrationof vapor may exist closer to the heat source, with a lower concentrationof vapor at the condenser end(s) of the heat pipe. The result is a massflow from one end to the other end of the member, taking advantage ofthe latent thermal capacity of the vapor to carry heat from one end tothe other. In one implementation, a thermal spreader with multiple heatpipes or multiple heat pipe sections may be fabricated by drillingappropriately-sized holes in a block of thermally conductive material,such as copper or aluminum, after which cylindrical-shaped heat pipesmay be inserted. Note, however, that various other thermal spreaderconfigurations utilizing heat pipes may be employed.

In FIG. 13, thermal modeling results are depicted comparing (within thecontext of a warm-liquid-cooling approach) a cold plate with a thermalspreader (that is either detachable or embedded within the cold plate),versus a cold plate without such a thermal spreader. The thermalperformance of two such cold plates is charted for a range of flow ratesof coolant (with water being used for comparison) to cool, by way ofexample, a 130 Watt electronic component. The temperature of theelectronic component was monitored at the center of the component, andas shown, the cold plate with the thermal spreader (having embedded heatpipes) outperforms the cold plate without the thermal spreader, with theperformance gain being approximately 0.05° C./W.

As explained further below, another advantage of the cooling apparatusesdisclosed herein is that a single cold plate design can be used toaccommodate different power maps without changing the design of the coldplate. For example, a cold plate designed for a six-core integratedcircuit chip could also be used for a dual core chip with higherfrequency, even though the power maps may be different. The coolingapparatuses disclosed herein can also accommodate different packagesizes, and different heat fluxes. For instance, the same cold platedesign could be used for multiple generations of CPUs, even though lidsizes on the chips are different. Thus, the cooling apparatusesdisclosed herein enable a single cold plate design to be employed withdifferent footprint sizes that can be used across multiple platforms,which makes use of the cooling apparatus straightforward. The designspresented herein also allow greater flexibility for locating the coldplate on the electronic component. That is, the electronic componentneed not be centered on the cold plate, and can be independent of thecoolant loop to the cold plate, which is not recommended for a coldplate without a thermal spreader such as disclosed herein due to thermalperformance loss. This advantage is illustrated in FIGS. 14A & 14B.

By way of example, FIG. 14A depicts (in bottom plan view) a coolingapparatus 1400 comprising a liquid-cooled cold plate 1410, shown coupledto an electronic component 1420 (such as a processor module) to becooled. Liquid-cooled cold plate 1410 includes a thermally conductivestructure 1411 and one or more coolant-carrying tubes 1415 extendingwithin thermally conductive structure 1411. As in the embodimentsdescribed above, coolant-carrying tube(s) 1415 comprises multiplecoolant-carrying channel sections 1406 disposed, in this example,substantially parallel to each other within thermally conductivestructure 1411. By off-centering electronic component 1420 from thecenter of the liquid-cooled cold plate, there is (shown by way ofexample) a single coolant-carrying channel section 1406 passing throughthe first region of the cold plate aligned to electronic component 1420.Thus, thermal transfer performance of this cooling apparatus might bedegraded from that of the cold plate described above in connection withFIGS. 11A & 11B.

FIG. 14B illustrates the advantage of incorporating a thermal spreaderwith embedded heat pipes as described herein between the electroniccomponent and the cold plate. Specifically, FIG. 14B illustrates acooling apparatus 1450 comprising a liquid-cooled cold plate 1410 (suchas described above in connection with FIG. 14A), and a thermal spreaderdisposed between electronic component 1420 and liquid-cooled cold plate1410. The thermal spreader includes multiple heat pipe sections 1425,some of which are shown disposed between electronic component 1420 andliquid-cooled cold plate 1410, notwithstanding off-centering of theelectronic component from the cold plate. These heat pipe sections 1425thus align at least partially to the above-described first region of theliquid-cooled cold plate, and partially align to the second region ofliquid-cooled cold plate. Thus, the thermal spreader with embedded heatpipe(s) facilitates lateral spreading of heat within the coolingapparatus, and thereby facilitates handling the electronic component tocold plate size differential, as well as offsetting of the electroniccomponent relative to the center of the cold plate, and is particularlybeneficial in a warm-liquid-cooling approach (such as described above).

FIG. 15A depicts an alternate embodiment of a cooling apparatus 1500, inaccordance with an aspect of the present invention. As shown, coolingapparatus 1500 comprises a liquid-cooled cold plate 1510 comprising oneor more coolant-carrying tubes 1512 extending within a thermallyconductive structure 1514 of liquid-cooled cold plate 1510. In oneexample, the one or more coolant-carrying tubes 1512 might comprisecopper tubes, and the thermally conductive material 1514 an aluminumblock, with the copper tubes being soldered within the aluminum block inone configuration. Cooling apparatus 1500 further includes a thermalspreader 1520 comprising one or more embedded heat pipes, such asdescribed above. In one embodiment, thermal spreader 1520 is an aluminumblock, within which the one or more heat pipes are embedded. Thermalspreader 1520 is (in one embodiment) detachably mounted to a thermalconduction surface 1511 of liquid-cooled cold plate 1510, for example,via an epoxy or brazed joint. As described above, cooling apparatus 1500detachably mounts to a surface 1531 of an electronic component 1530 (forexample, a processor module) to be cooled, and liquid-cooled cold plate1510 comprises a first region 1515 defined by a projection of where thesurface to be cooled 1531 aligns to the cold plate, and a second region1517 comprising that region of the cold plate outside first region 1515.As in the examples discussed above, thermal spreader 1520 includes oneor more heat pipes comprising one or more heat pipe sections disposed topartially align to first region 1515 of liquid-cooled cold plate 1510,and to partially align to second region 1517 of liquid-cooled cold plate1517, which facilitates lateral heat transfer from the electroniccomponent to coolant-carrying channel sections of liquid-cooled coldplate 1510 in the second region 1517 of the liquid-cooled cold plate.

FIG. 15B illustrates one detailed embodiment of thermal spreader 1520.In this embodiment, thermal spreader 1520 includes a thermallyconductive structure 1521, such as a metal plate or a block, withmultiple U-shaped heat pipes 1522 and a single straight heat pipe 1524embedded therein, which illustrates one pattern designed to laterallyspread heat outward from the electronic component into the cold plate.With the electronic component disposed as illustrated in FIG. 15A, eachheat pipe 1522, 1524 (FIG. 15B) at least partially aligns to the firstregion 1515 of the liquid-cooled cold plate, and to the second region1517 of the liquid-cooled cold plate, thereby facilitating transfer ofheat out to the coolant-carrying tube sections in second region 1517 ofthe liquid-cooled cold plate.

FIG. 15C illustrates one embodiment of a liquid-cooled cold plate 1510,wherein a single coolant-carrying tube 1512 is shown embedded withinthermally conductive structure 1514. In this example, a coolant inletport 1508 and coolant outlet port 1509 are illustrated as extending froma common side edge of liquid-cooled cold plate 1510. The placement ofthe ports may be optimized depending on the specific coolingapplication. As explained above, a liquid-cooled cold plate 1510 such asillustrated in FIG. 15C is substantially less costly to manufacture thana micro-channel cold plate, such as illustrated in FIG. 10A.

Note that in the embodiments of FIGS. 15A-15C, thermal conductionsurface 1511 of liquid-cooled cold plate 1510 is larger than the surface1531 to be cooled of electronic component 1530, and (in this example)the surface area of the main, opposite sides of thermal spreader 1520are substantially the same as that of thermal conduction surface 1511 ofliquid-cooled cold plate 1510. As noted, thermal spreader 1520 may bedetachable or integrated with the liquid-cooled cold plate structure.Note that, in one example, the surface area of thermal conductionsurface 1511 may be two times (or more) larger than that of the surface1531 to be cooled of the electronic component. If desired, appropriatealignment features may be provided on the liquid-cooled cold plate andthe thermal spreader to facilitate aligning the thermal spreader to theliquid-cooled cold plate, and/or to the electronic component. Note thatthe particular heat pipe(s) configuration and layout within the thermalspreader, as well as the particular coolant-carrying tube(s)configuration and layout within the cold plate, may be optimized for aparticular application. In most applications, the heat pipes embeddedwithin the thermal spreader cover a significant portion of the footprintof the thermal spreader.

FIGS. 16A-16C depict various alternate embodiments of the coolingapparatus of FIGS. 15A-15C.

In FIG. 16A, a cooling apparatus 1600 is depicted with a thermalspreader 1520 coupled to a main surface 1601 of the liquid-cooled coldplate 1510′ on an opposite side of the cold plate from thermalconduction surface 1511, meaning that, in this embodiment, the coldplate is disposed between electronic component 1530 and thermal spreader1520. Also note that, by way of example, the liquid-cooled cold plate1510′ comprises coolant inlet port 1508 and coolant outlet port 1509disposed at opposite side edges of the liquid-cooled cold plate.

In FIG. 16B, a cooling apparatus 1600′ is illustrated similar to thecooling apparatus described above in connection with FIG. 15A, but withthe addition of a second thermal spreader 1620 coupled to main surface1601 of liquid-cooled cold plate 1510 on the opposite side of the coldplate from the thermal conduction surface. By way of example, the firstthermal spreader 1520 and second thermal spreader 1620 each have one ormore heat pipes embedded therein with one or multiple heat pipe sectionspartially aligned to the first region 1515 of liquid-cooled cold plate1510, and at least partially aligned to the second region 1517 of theliquid-cooled cold plate.

In FIG. 16C, the cooling apparatus 1500 of FIG. 15A is shown coupled tomultiple electronic components 1530. By way of example, coolingapparatus 1500 might couple to two or more electronic components, suchas two or more processor modules. Thus, the cooling apparatus disclosedherein (with the electronic module to cold plate size differential),allows greater flexibility for use of the same cold plate design to coolmultiple electronic components of an electronic system.

FIGS. 17A-17D depict various embodiments of a thermal spreader 1700 of acooling apparatus such as described above in connections with FIGS.12A-12C & 14B-16C. In these figures, thermal spreader 1700 is coupled toan electronic component 1710 to be cooled. In FIG. 17A, three heat pipes1720 are shown embedded within the thermal spreader. These heat pipes1720 are straight heat pipes, shown substantially parallel to eachother, and of sufficient size and spacing to extend partially over thesurface to be cooled of electronic component 1710, and partially outsideof the surface to be cooled, thereby facilitating lateral spreading ofheat from the region over the surface to be cooled outward within thecooling apparatus, as described above.

In the embodiment of FIG. 17B, multiple U-shaped heat pipes 1730 aresubstituted for the multiple straight heat pipes 1720 of the coolingapparatus embodiment of FIG. 17A. In this embodiment, thermal spreader1701 is shown coupled to electronic component 1710 to be cooled tofacilitate lateral spreading of heat within the cooling apparatus fromthe electronic component outward to regions of the cooling apparatusdisposed outside a projection of the surface to be cooled into thecooling apparatus.

In FIG. 17C, a thermal spreader 1702 is illustrated comprising multipleU-shaped heat pipes 1730, and a single straight heat pipe 1720. In thisembodiment, U-shaped heat pipes 1730 and straight heat pipe 1720 aresized and configured to again extend partially over the surface to becooled of electronic component 1710 and partially into the second regionof the cooling apparatus disposed outside of the first region.

In FIG. 17D, a thermal spreader 1703 is illustrated comprising multiplesmaller, U-shaped heat pipes 1730 and a single straight heat pipe 1720.As with the above-described embodiments, portions of these heat pipesalign over the surface to be cooled, and extend into the region of thecooling apparatus outside of the surface to be cooled, therebyfacilitating lateral transfer of heat within the cooling apparatus.

Note that the above-described embodiments are provided by way ofexample, only. The integration of high-efficiency cold plates andthermal spreading employing heat pipes significantly enhances thethermal transfer effectiveness of the cold plate design. Advantageously,in the cooling apparatuses disclosed herein, a common design may beemployed to cool different electronic components, notwithstandingdifferent component locations, sizes and heat fluxes. This reduces thenumber of parts required to implement a liquid-cooled cooling approach,and thereby provides benefit to the supply chain. That is, a singlecooling apparatus part may be employed to facilitate liquid-cooling ofmultiple different electronic components.

Although embodiments have been depicted and described in detail herein,it will be apparent to those skilled in the relevant art that variousmodifications, additions, substitutions and the like can be made withoutdeparting from the spirit of the invention and these are thereforeconsidered to be within the scope of the invention as defined in thefollowing claims.

What is claimed is:
 1. A method of facilitating dissipation of heat fromat least one electronic component, the method comprising: providing acooling apparatus configured to couple to at least one electroniccomponent to be cooled, the cooling apparatus having a larger footprintsize than the at least one electronic component to be cooled, and thecooling apparatus comprising: a liquid-cooled cold plate comprising athermally conductive material with a plurality of coolant-carryingchannel sections extending therein, the liquid-cooled cold platecomprising a thermal conduction surface having a first surface area, andwherein the at least one electronic component comprises a surface to becooled, the surface to be cooled comprising a second surface area,wherein the liquid-cooled cold plate is oversized relative to the atleast one electronic component, with the first surface area of thethermal conduction surface being greater than the second surface area ofthe surface to be cooled and in operation, heat is transferred from thesurface to be cooled of the at least one electronic component to thethermal conduction surface of the liquid-cooled cold plate, and theliquid-cooled cold plate comprises a first region where the surface tobe cooled aligns to the liquid-cooled cold plate and a second regionoutside the first region, and wherein at least one firstcoolant-carrying channel section of the plurality of coolant-carryingchannel sections is disposed, at least in part, within the first region,and at least one second coolant-carrying channel section of theplurality of coolant-carrying channel sections of the liquid-cooled coldplate is disposed in the second region, outside the first region, andlaterally offset from the electronic component; at least onecoolant-carrying tube, the at least one coolant-carrying tube extendinginto, through and out from, the liquid-cooled cold plate and beingembedded within the thermally conductive material at the thermalconduction surface where extending through the liquid-cooled cold plate,the at least one coolant-carrying tube being exposed at the thermalconduction surface of the liquid-cooled cold plate so as to be coupledto a first main surface of a first thermal spreader, the at least onecoolant-carrying tube comprising the plurality of coolant-carryingchannel sections; the first thermal spreader being disposed between theat least one electronic component and the liquid-cooled cold plate, andhaving the first main surface and an opposing second main surface, thefirst main surface of the first thermal spreader being coupled to thethermal conduction surface of the liquid-cooled cold plate, and thefirst thermal spreader being a block of thermally conductive materialwith a plurality of heat pipes disposed therein, each heat pipecomprising multiple heat pipe sections, at least one heat pipe sectionof the multiple heat pipe sections being partially aligned to the firstregion of the liquid-cooled cold plate, between the at least oneelectronic component and the at least one first coolant-carrying channelsection of the liquid-cooled cold plate, and partially aligned to thesecond region of the liquid-cooled cold plate, below the at least onesecond coolant-carrying channel section of the liquid-cooled cold plate,laterally away from the at least one electronic component, the pluralityof heat pipes of the first thermal spreader facilitating distributingheat laterally outward from the at least one electronic component to theat least one second coolant-carrying channel section of theliquid-cooled cold plate in the second region of the liquid-cooled coldplate, and wherein the plurality of heat pipes are discrete heat pipestructures inserted into the block of thermally conductive material ofthe first thermal spreader so that the heat pipes physically contacteach other lengthwise between the at least one electronic component andthe first region of the liquid-cooled cold plate; and a second thermalspreader, the second thermal spreader being coupled to a main surface ofthe liquid-cooled cold plate, the main surface of the liquid-cooled coldplate and the thermal conduction surface of the liquid-cooled cold platebeing opposite sides of the liquid-cooled cold plate; and wherein thefirst thermal spreader and the liquid-cooled cold plate areco-extensive.
 2. The method of claim 1, wherein the first thermalspreader and the liquid-cooled cold plate are each integrated within acommon, thermally conductive structure.
 3. The method of claim 1,wherein the first region of the liquid-cooled cold plate is offset froma center of the liquid-cooled cold plate.
 4. The method of claim 1,wherein the at least one electronic component comprises multipleelectronic components, including a first electronic component and asecond electronic component, and wherein both the first electroniccomponent and the second electronic component are coupled to the firstthermal spreader.
 5. The method of claim 1, wherein a first heat pipesection of the multiple heat pipe sections of the first thermal spreaderis a U-shaped heat pipe section.
 6. The method of claim 5, wherein asecond heat pipe section of the multiple heat pipe sections of the firstthermal spreader is a straight heat pipe section.
 7. The method of claim1, further comprising: providing a coolant loop coupled in fluidcommunication with the plurality of coolant-carrying channel sections ofthe liquid-cooled cold plate; and providing an outdoor-air-cooled heatexchange unit coupled to facilitate heat transfer from the liquid-cooledcold plate to the outdoor-air-cooled heat exchange unit via, at least inpart, the coolant loop, the outdoor-air-cooled heat exchange unitcooling coolant passing through the coolant loop by dissipating heatfrom the coolant to outdoor ambient air.
 8. A method comprising:providing at least one electronic component; and providing a coolingapparatus coupled to the at least one electronic component fordissipating heat from the at least one electronic component, the coolingapparatus having a larger footprint size than the at least oneelectronic component, and comprising: a liquid-cooled cold platecomprising a thermally conductive material with a plurality ofcoolant-carrying channel sections extending therein, the liquid-cooledcold plate comprising a thermal conduction surface having a firstsurface area, and wherein the at least one electronic componentcomprises a surface to be cooled, the surface to be cooled comprising asecond surface area, wherein the liquid-cooled cold plate is oversizedrelative to the at least one electronic component, with the firstsurface area of the thermal conduction surface of the liquid-cooled coldplate being greater than the second surface area of the surface to becooled of the at least one electronic component, and in operation, heatis transferred from the surface to be cooled of the at least oneelectronic component to the thermal conduction surface of theliquid-cooled cold plate, and the liquid-cooled cold plate comprises afirst region where the surface to be cooled aligns to the liquid-cooledcold plate and a second region outside the first region, and wherein atleast one first coolant-carrying channel section of the plurality ofcoolant-carrying channel section is disposed, at least in part, withinthe first region, and at least one second coolant-carrying channelsection of the plurality of coolant-carrying channel sections of theliquid-cooled cold plate is disposed in the second region, outside thefirst region, and laterally offset from the electronic component; atleast one coolant-carrying tube, the at least one coolant-carrying tubeextending into, through and out from, the liquid-cooled cold plate andbeing embedded within the thermally conductive material at the thermalconduction surface where extending through the liquid-cooled cold plate,the at least one coolant-carrying tube being exposed at the thermalconduction surface of the liquid-cooled cold plate so as to be coupledto a first main surface of a first thermal spreader, the at least onecoolant-carrying tube comprising the plurality of coolant-carryingchannel sections; the first thermal spreader being disposed between theelectronic component and the liquid-cooled cold plate, and having thefirst main surface and an opposing second main surface, the first mainsurface of the first thermal spreader being coupled to the thermalconduction surface of the liquid-cooled cold plate, and the firstthermal spreader being a block of thermally conductive material with aplurality of heat pipes disposed therein, each heat pipe comprisingmultiple heat pipe sections, at least one heat pipe section of themultiple heat pipe sections being partially aligned to the first regionof the liquid-cooled cold plate, between the electronic component andthe at least one first coolant-carrying channel section of theliquid-cooled cold plate, and partially aligned to the second region ofthe liquid-cooled cold plate, below the at least one secondcoolant-carrying channel section of the liquid cold plate, laterallyaway from the electronic component, the plurality of heat pipes of thefirst thermal spreader facilitating distributing heat laterally outwardfrom the electronic component to the at least one secondcoolant-carrying channel section of the liquid-cooled cold plate in thesecond region of the liquid-cooled cold plate, and wherein the pluralityof heat pipes are discrete heat pipe structures inserted into the blockof thermally conductive material of the first thermal spreader so thatthe heat pipes physically contact each other lengthwise between theelectronic component and the first region of the liquid-cooled coldplate; a second thermal spreader, the second thermal spreader beingcoupled to a main surface of the liquid-cooled cold plate, the mainsurface of the liquid-cooled cold plate and the thermal conductionsurface of the liquid-cooled cold plate being opposite sides of theliquid-cooled cold plate; and wherein the first thermal spreader and theliquid-cooled cold plate are co-extensive.
 9. The method of claim 8,wherein the first region of the liquid-cooled cold plate is offset froma center of the liquid-cooled cold plate.